The one-paragraph version
Text-to-SVG generation turns a written description into a scalable vector image — code made of shapes, curves and colors rather than pixels — but there has been no good way to judge whether the result actually matches the description. This paper introduces SVG-Score, a human-grounded way to score that match. The authors first prove that the current standards fail: the common CLIPScore barely notices when colors, counts or left-right relations are wrong, and off-the-shelf vision-language models used as judges notice more but very unevenly, especially on black-and-white drawings. They then collect about 13,000 human ratings of how well an SVG matches its caption on a 1-to-5 scale, and use those ratings to train two complementary judges: fast CLIP-based scorers first adapted to vector graphics and then tuned to human preferences, and a slower, explainable Qwen3-VL-8B judge tuned with supervised learning plus reinforcement learning to reproduce both absolute scores and relative rankings. Both new judges agree with humans substantially better than all existing baselines, and when used to benchmark 16 current generators they show commercial large language models far ahead, with quality collapsing for almost all systems as prompts get compositional.
The problem they're solving, and why it matters
Scalable Vector Graphics, meaning images stored as instructions like draw a blue circle here rather than as a grid of pixels, are central to logos, icons, fonts and illustrations because they can be scaled, edited and stored compactly.
Generation of these graphics from text is advancing through two routes: optimization-based methods that iteratively adjust curves at test time by back-propagating an image loss through a differentiable rasterizer, and autoregressive methods that treat SVG code as text and have a language model write it out directly.
Evaluation has not kept up. Practically everyone reuses metrics built for natural photos, most notably CLIPScore, meaning the cosine similarity between CLIP image and text embeddings. CLIP, meaning Contrastive Language-Image Pre-training, was never trained on sparse, abstract vector art with flat colors and parametric structure, so it faces a distribution shift. Similarly, people now prompt a Vision-Language Model, meaning a model that takes both image and text, to act as a judge with a rubric, but those judges are used zero-shot without ever being trained to match human judgments on SVGs. Without a trusted, SVG-specific measure of Semantic Alignment, meaning how faithfully the rendered image reflects objects, attributes, counts, spatial relations and overall meaning in the caption, the field cannot tell real progress from good-looking misses.
The key idea, in plain words
If you want a judge that thinks like a person looking at vector art, you have to show it what people actually penalize and then explicitly teach it that behavior at two speed-accuracy points.
The authors do this in three moves. First, they stress-test existing judges with controlled corruptions to make the blind spots visible. For example, keep a pink scale drawing fixed but change the caption from pink to green, from four buttons to five, or from buttons below the display to above it — a sensitive judge should drop its score. Second, they build the missing supervision: thousands of caption-SVG pairs where people give a holistic 1-to-5 Semantic Alignment score, covering unrelated to faithful. Third, they distill that supervision into two tools that share the same human notion of correctness: a lightweight pair of CLIP encoders re-tuned for SVG look and human taste for cheap large-scale ranking, and a heavier vision-language judge that writes a short rationale plus a score for careful, interpretable evaluation.
How it actually works, step by step
1. Prove the failure with perturbations. They sample 200 SVGs whose captions mention a color, a spatial relation and a quantity. They keep the image fixed and swap one meaning element at a time — for instance each mentioned color replaced by five alternatives and averaged, a spatial word replaced by its opposite, a number replaced by nearby and farther numbers. As coarse controls they also keep the caption fixed and replace the image with a crude proxy: one colored circle per mentioned color, or a solid background in a mentioned color. Sensitivity is measured in words as score on corrupted pair minus score on original pair. Negative means correctly penalized; near zero means blind.
2. Collect human Semantic Alignment data. Starting from OmniSVG captions, they retrieve six additional similar-but-not-identical SVGs per caption to get a spread from good to unrelated matches. Five annotators see a caption plus rendered SVGs and assign a single holistic score using a defined scale where 1 means unrelated or nearly unrelated and 5 means semantically faithful, judging objects, color and shape attributes, layout and counts together, not as separate subscores. This yields 12,957 ratings over 8,671 unique SVGs and 1,858 unique captions. Crucially they split by keeping no caption and no exact SVG string in both train and test, leaving 10,583 training and 2,374 test ratings, and they check reliability by triple-rating 100 test items.
3. Build a separate generator benchmark. Independently they create 1,616 new captions at three levels — 1,000 Easy simple descriptions, 500 Medium, 116 Hard compositional prompts with multiple objects, attributes, counts and relations — written by three vision-language captioners and human-validated. This set is never used for training, only for testing generators.
4. Train the fast judge: SVG-adapted CLIP. They take three CLIP backbones, ViT-B/32, ViT-L/14 and ViT-H/14, where ViT means Vision Transformer and the numbers denote patch size and model size. They first continue contrastive training on about 3.2 million SVG-caption pairs from StarVector and OmniSVG after recaptioning noisy originals with Qwen3-VL-8B. They then freeze that model and train small LoRA adapters, meaning Low-Rank Adaptation matrices that add a tiny trainable update to frozen weights, on human preference pairs: for the same caption, the higher human-scored SVG should get higher text-image similarity under a two-way choice loss.
5. Train the expressive judge: human-aligned VLM. They start from Qwen3-VL-8B and do supervised fine-tuning for one epoch to output a fixed structure with brief reasoning followed by a score. They then run GRPO, meaning Group Relative Policy Optimization, a reinforcement learning method that compares multiple sampled outputs for the same input, with two rewards: an ordinal reward that gives full credit for the exact human score and decreasing credit or penalty as the prediction moves 1 to 4 steps away, plus a small format bonus, and an intra-caption ranking reward that rewards correctly ordering two different SVGs for the same caption when humans ordered them differently. Rendering failures get the minimum score.
What they showed — results, honestly, with limitations
Off-the-shelf judges are insensitive in exactly the ways that matter. Vanilla CLIP ViT-B/32 shows near-zero change for spatial and count swaps and only a small change for color swaps, but a large drop when the image is replaced by circles or a blank background. In other words it sees gross visual loss but not compositional errors. Zero-shot Qwen3-VL-8B is more sensitive on color SVGs but its response to counts and spatial relations largely disappears on black-and-white graphics.
Domain adaptation plus preference tuning both help CLIP. On the held-out human test, agreement measured by rank correlations and pairwise accuracy, meaning how often the model orders two SVGs for the same caption the same way humans did, rises at each stage. For example on ViT-B/32 Spearman rank correlation, meaning agreement on ordering from best to worst, goes from about 42.9 to 54.3 after SVG adaptation to 59.3 after preference alignment. The best CLIP, ViT-H/14, reaches 63.2 and 80.6% pairwise accuracy, beating the strongest natural-image preference scorer HPSv2 by about 8 and 5 points at equal size. Honestly, it does not win everywhere: ImageReward retains better absolute error, and the mid-size ViT-L/14 has slightly higher linear Pearson correlation than ViT-H/14, suggesting the largest model orders better but calibrates absolute numbers less well.
The trained VLM judge is the best overall. Zero-shot Qwen3-VL-8B already beats all CLIP scorers at 67.8 rank correlation, ahead of Claude Haiku 4.5, GPT-5.4-mini and VectorGym, the only prior SVG-specific evaluator, at 65.1 — showing SVG task competence alone does not equal human agreement. Their trained judge rises to 74.9 rank correlation and lowers mean absolute error from 0.80 to 0.68, best on every reported metric, with the narrowest lead on pairwise ordering. Ablations show neither stage alone suffices: supervised tuning alone hurts rank correlation but helps calibration, reinforcement learning alone helps rank but hurts calibration, together they jump well beyond either.
Benchmarking 16 generators shows a clear hierarchy and fragility. Commercial systems lead: Claude Sonnet 5 is first under six of seven evaluators, GPT-5 nano first under the remaining CLIP-H. Among others, HiVG leads open-source models, while painterly VectorFusion gets the highest VLM score of any non-commercial system. Rendering reliability varies wildly, from near zero failures for commercial and painterly methods to 33% for IntroSVG. Performance collapses with complexity — only Claude Sonnet 5 holds up on Hard prompts — and CLIP-based scores can be misleading, for example ranking abstract CLIPDraw strokes second on Hard captions because CLIPDraw directly optimizes a CLIP objective, while the new VLM judge keeps it near the floor.
Limitations the authors state plainly. They measure only rendered semantic match, not editability, path efficiency, layer organization or code quality. Most human pairs have a single rating, so estimates are noisy despite good annotator agreement. The generator benchmark itself has no human ratings; its ranking relies on automatic judges validated elsewhere. Learned judges may fail on styles far from training.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, the perturbation protocol is a reusable template: systematically break color, count and relation words and test whether scores move, separately for sparse black-and-white versus color inputs where the paper finds judges go blind. That points directly to where compositional grounding still fails.
For world models and 3D generation and scene representation, the lesson is that photo-trained scorers detect that something is present but not that it is correctly arranged — the same failure that would let a world model place objects in wrong relations yet score highly. The two-reward design here, one reward for absolute correctness and one for correctly ordering alternatives for the same prompt, is directly portable to training reward models for layout, physics plausibility or view consistency.
For text-to-CAD and parametric geometry, the parallel is closest: like SVG, CAD is code for precise primitives where a small parameter error breaks function but barely changes pixels. A fast adapted embedding scorer for large-scale search plus a slower rationale-producing judge for final validation, both grounded in human ratings of semantic fidelity rather than photorealism, is a practical evaluation pattern to copy, while remembering this work does not yet judge editability or parametric cleanliness that CAD workflows require.
Terms worth knowing
Scalable Vector Graphics (SVG): image format that stores shapes as code such as paths and Bezier curves, infinitely scalable and editable, unlike raster images that store pixel grids.
Semantic Alignment: the paper's core human score from 1 meaning unrelated to 5 meaning faithful, judging whether objects, attributes like color and size, counts, spatial relations and overall meaning in the caption appear in the rendering.
CLIPScore: reference-free metric that scores image-text match as similarity in CLIP embedding space; fast but trained on natural photos, not vector art.
VLM-as-a-Judge: using a vision-language model prompted with criteria and a 1-to-5 rubric to score faithfulness, attractive for long prompts but uneven off-the-shelf.
Perturbation sensitivity: testing a judge by corrupting one meaning element and checking that its score drops; near-zero change reveals blindness to that error type.
Supervised fine-tuning (SFT) and GRPO: first teaching the judge the required output format by imitation, then improving it with reinforcement learning that compares sampled outputs using ordinal closeness to the human score and correct relative ordering.
LoRA adapters: tiny trainable matrices added to a frozen model to adapt it cheaply to a new domain.
Correlation and pairwise accuracy: Spearman, Pearson and Kendall measures of how well predicted scores track human scores in rank order or linear value, and how often the model orders two SVGs for the same caption the way humans did.