The one-paragraph version
This paper tackles a frustration with machine-made vector graphics: current systems can copy how an image looks, but they produce a flat pile of disconnected curves with no understanding that some curves form a crown, some form a head, and so on. The authors build an agent-like pipeline where a Vision-Language Model — meaning a model that reads both images and text — acts as a scene parser that repeatedly asks, can this region be split into named parts, grounds those names to pixels, cleans up leftovers, and then repairs what is hidden behind foreground objects so each part is whole. Each whole part is then turned into vector code and nested inside labeled groups that mirror the parsing tree. To prove this matters they also release a new test set of 203 human-grouped graphics plus new measures of whether groups match human meaning and whether deleting an object leaves a clean background, and they show their predicted groups beat even generously optimized versions of prior flat methods on grouping and editability while keeping image fidelity high.
The problem they're solving, and why it matters
Scalable Vector Graphics, usually shortened to SVG, is a way to describe an image not as a grid of pixels but as editable mathematical shapes like paths, rectangles, and curves, organized in a text file. That editability is the whole point: a designer should be able to select the crown and delete it, move the tram, or recolor the sea.
The paper argues existing SVG generation misses this. Optimization-based tracers, meaning programs that cluster similar-colored pixels into paths, and direct VLM generators, meaning models that write out SVG code token by token, both output what the authors call flat and semantically agnostic collections. Flat here means all paths sit at the same level with no folders. Semantically agnostic means the file does not know which paths belong to which human concept.
That causes two concrete failures, illustrated in Figure 1 and Figure 3. First, entanglement: a tracer may merge the similarly colored tram and background into one path, so you cannot select one without the other. Second, holes: if you delete the foreground tram, the background behind it was never drawn, so you see a white cutout instead of continuous scenery. Trying to fix grouping after the fact is bounded by those bad paths — you cannot un-merge an entangled path or invent missing hidden geometry.
Existing evaluations hide the problem because they only measure whole-image likeness, meaning does the render look like the input overall. They never test whether parts are meaningful or editable.
The key idea, in plain words
Instead of generating vectors first and hoping meaning emerges, parse meaning first and generate vectors only for meaningful, complete pieces.
Think of how a human illustrator works in layers, back to front. They would draw the full sea even where sand will later cover it, then draw sand on top, then draw the parasol on top of that. The file keeps those as separate named layers in painter's order.
The authors make a VLM do that job as a manager with four roles, calling specialist tools for the pixel work. The VLM decides what the named parts are and in what back-to-front order they stack, checks its own work for missed scraps and for occlusion, and guesses the full hidden outline so an image inpainter can fill it in. Only after every leaf piece — meaning a final part that should not be split further — is isolated and made whole does the system convert pixels to vector paths and wrap them in nested group tags like <g id="sea"> that directly reflect the parse tree.
A second key idea is to measure what they claim: a human-annotated benchmark where annotators grouped raw paths into self-contained entities that keep their identity when separated, plus two families of metrics that test grouping and actual deletion, not just looks.
How it actually works, step by step
The input is normally a raster image, meaning a regular pixel image, which becomes the root of a semantic tree. A semantic tree here means a hierarchy where each node is a named visual component and edges mean part-of, for example scene contains beach contains sand.
First, top-down decomposition. For each active node, the VLM acting as Hierarchical Decomposer sees the full image plus text context and answers a single question: is this semantically atomic or decomposable. Atomic means it should not be split further, like a laptop or a head under their strict prompt rules. If decomposable, the VLM outputs child labels and boxes in back-to-front order. The prompt given in the appendix is very conservative: it defaults to atomic, requires each child to cover at least 25 percent of the parent box, requires visible occlusion between parts, and almost always stops at depth two.
Second, semantic grounding. Grounding here means turning a word plus rough box into exact pixels. The predicted labels and boxes are passed to Segment Anything Model 3, shortened to SAM 3, which returns pixel masks for each concept.
Third, residual judging and recursion. Because masks rarely cover every pixel cleanly, leftover pixels are collected into a residual image. The VLM acting as Residual Judge decides if that leftover is a missed concept to add as a new node or just noise slivers and dots to discard. The system then repeats the split-ground-judge loop breadth-first until all branches end in atomic leaves.
Fourth, occlusion recovery for leaves. Cutting out foreground naturally leaves holes in background. The VLM acting as Occlusion Assessor looks at each isolated leaf versus the full scene and decides if a real chunk is cut out, not just edge noise. If yes, the VLM acting as Spatial Estimator predicts a closed polygon with 30 to 50 vertices for the amodal shape. Amodal means the complete shape including hidden parts, as opposed to only the visible fragment. The inpainting mask is then the difference between that predicted full polygon and the visible SAM mask. That mask plus the leaf image and label goes to an inpainting model, Flux-Fill, to synthesize the missing geometry conditioned on the concept.
Fifth, vectorize and assemble. Each completed leaf image is converted to mathematical primitives with the external tracer VTracer. Then the tree is compiled bottom-up into SVG: leaf primitives go inside labeled group tags, intermediate nodes become grouping tags that contain their children, all ordered background to foreground to respect z-order, meaning depth stacking.
For Text-to-SVG, meaning generating SVG from a text prompt rather than an image, they simply cascade: first generate a raster with a fine-tuned text-to-image diffusion model, FLUX.1-dev or SD3.5-medium with low-rank adapters trained on vector-style data, then feed that raster through the same image-to-SVG pipeline.
What they showed — results, honestly, with limitations
They evaluated Image-to-SVG on their new Semantic SVG Benchmark of 203 samples covering icons, illustrations, and emoji, with on average about 2.6 leaf parts per image and about 7.5 raw paths per leaf. Annotator agreement was reported as very high.
For grouping they distinguish two settings. Predicted means they test the model's own groups. Optimal means for flat baselines with no groups, they generously search for the subset of generated paths that best reconstructs each human group, using a greedy search, to give an upper bound. Even with that help, their natively predicted groups win. For example, grouping recall measured with pixel error, where lower is better, is 0.0174 for theirs predicted versus 0.0227 for the strongest baseline VTracer optimal and much worse for OmniSVG, InternSVG, StarVector, and prompted general VLMs. With a semantic similarity measure based on DINOv2 features, where higher is better, theirs is 0.9487 versus 0.9392 for VTracer optimal. Whole-image fidelity stays competitive: theirs 0.0049 pixel error and 0.9845 DINO similarity versus VTracer 0.0036 and 0.9922, far ahead of generative baselines.
For functional editability they introduce Post-Edit Rendering Error, shortened to PERE, which deletes a front occluder from both generated and human SVG and compares the remaining renders over the full frame. It punishes both wrong grouping and missing hidden geometry. Their predicted PERE of 0.0210 is about 47.5 percent lower than VTracer optimal at 0.0400, and ahead of all baselines even in optimal form. A small human study moving and removing objects also found all 15 edits completed with their files versus 11 of 15 and 13 of 15 for VTracer, roughly 1.9 times faster, and much lower error after removal, 0.006 versus 0.023.
Ablations show the VLM backbone's spatial accuracy matters: Gemini-3-flash as default beats Claude-Sonnet-4.6, GPT-5, and Qwen3.6-35B-A3B, with GPT-5 often under-segmenting and Qwen struggling with box localization. Removing inpainting hurts grouping and raises PERE by about 31.6 percent. Text-to-SVG on 300 MMSVG-Bench prompts shows best text-alignment and aesthetic scores for their FLUX variant, though OmniSVG gets the lowest FID distance, which authors describe as an artifact because that FID reference comes from OmniSVG's own training distribution.
Limitations are stated plainly. Cost and speed are high: about 16.3 VLM calls and 255 seconds per image on average, 94.9 percent spent on sequential inpainting, costing about 21 cents per image. The benchmark trees are shallow, maximum depth 2 and mean depth 1.02, so deep hierarchy is untested. The optimal baseline is only a greedy approximation, not a certified global optimum, and PERE is not normalized by base reconstruction error, so it mixes generation and grouping quality.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this is a demanding probe beyond bounding boxes or captions: the model must propose part names, order them back-to-front, judge residual scraps, and draw a 30 to 50 point polygon for what it cannot see. Success tracks closely with the backbone's localization ability, which suggests compositional vectorization could be a useful benchmark for grounded spatial reasoning, and failures like merging colors or missing occluders reveal where current VLMs lack object persistence.
For world models, meaning internal models that predict how scenes are structured and how they change under actions, the amodal completion step is directly relevant. The system explicitly represents that the sea continues behind the sand, so a delete or move action leaves a plausible world rather than a hole. That object-centric, occlusion-aware layering is a 2D analogue of maintaining a persistent scene state.
For text-to-CAD and parametric geometry, the shift from pixels to named, complete, ordered primitives matters. CAD needs selectable sketches and extrusions, not pixel clusters. This pipeline's pattern — decompose semantically, estimate full parametric outline, then emit editable code with hierarchy — maps naturally to decomposing a mechanical part into base plate plus bosses and holes, each geometrically closed and separately editable, even if the current primitives are only flat color paths.
For 3D generation and scene representation, the lesson is that visual fidelity alone does not guarantee usable structure. Just as a 3D mesh soup or splat cloud can look right but be uneditable, a flat SVG can look right but fail under editing. Their metrics for semantic recall and post-edit error, plus the idea of assembling leaves back-to-front with recovered hidden geometry, point toward layered or grouped 3D outputs where each object is whole, labeled, and manipulable.
Terms worth knowing
- SVG: Scalable Vector Graphics. An image format written as text code describing shapes as math, so it stays sharp at any size and can be edited by selecting shapes.
- Semantic compositional SVG: An SVG where paths are grouped into named nested groups that correspond to human concepts and part-whole relations, not just a flat list.
- VLM: Vision-Language Model. A model that takes both images and text, here used to reason about what parts exist and how they stack.
- Semantic tree: The hierarchy the VLM builds, where the root is the whole scene, intermediate nodes are groups, and leaves are final atomic parts.
- Grounding: Converting a word and rough box into exact pixels, here done with SAM 3.
- Residual: Leftover pixels not claimed by any predicted part, which the system must either promote to a new part or discard as noise.
- Amodal shape: The full shape of an object including parts hidden behind something else, versus only the visible fragment.
- Occlusion and inpainting: Occlusion is when a front object covers a back object. Inpainting here synthesizes the covered region so the back object is complete.
- Z-order / painter's order: Back-to-front stacking order that determines what covers what when rendered.
- Semantic Recall and Precision: Recall asks for each human group, did the system produce something that looks like it. Precision asks for each system group, does it look like some human group, penalizing nonsense groups.
- PERE, Post-Edit Rendering Error: A functional test that deletes a front object from both generated and human SVG and measures the difference in what remains, catching bad grouping and holes at once.