All summaries · 2026-10-21

On the Limits of Scale: Inverse Scaling in Vision-Language Models

Rohit Saxena, Alessandro Suglia, Marek Rei, et al.

arXiv:2610.27360 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-10-21

The one-paragraph version

This paper asks a simple question that cuts against the usual story of AI progress: do bigger vision-language models always get better at everything? The authors test 19 open models from four families — Qwen3-VL, InternVL3.5, Gemma 3, and Molmo 2, ranging from about 1 billion to 235 billion total parameters — on eight standard multimodal benchmarks. In total they run 29,386 questions per model, or 558,334 model-question evaluations. At the level of whole-benchmark averages, bigger is usually better. But when they look one level down, at named subtasks like “Jigsaw” or “Multi-view Reasoning” in BLINK or “Organic Chemistry” in MMMU, they repeatedly find the opposite: a smaller model in the same family beats a larger one on that same set of questions. They count 375 such subtask-and-model-pair cases, covering 4,342 unique questions where the small model is right and the large model is wrong. Thirteen subtask types show this pattern in three or four independent families, which suggests it is driven by the kind of task, not just one buggy checkpoint. The hardest-hit areas are multi-image spatial reasoning, basic visual perception, and university-level science with diagrams, with gaps over 20 percentage points in several cases.

The problem they're solving, and why it matters

The field has relied on a working rule called scaling laws: give a model more parameters, more data, and more compute, and performance should go up in a smooth, predictable way.

That rule mostly holds for averages. A 32-billion-parameter model will usually beat its 4-billion-parameter sibling on a whole benchmark. The problem is that an average can hide regressions. A model can gain 5 points overall by improving on common questions while quietly losing 20 points on a rare but important skill.

Earlier work had already shown this kind of failure, called inverse scaling, in text-only language models: bigger models did worse on certain trick questions because they leaned harder on strong priors, memorized distractors, or imitated unhelpful patterns. A related line showed that giving a reasoning model more thinking time could also make it worse. And a concurrent study showed that adding visual distractors could induce inverse-scaling-like drops in vision-language models, but in a deliberately constructed setting.

What was missing was a systematic check under normal conditions: do current vision-language models, evaluated on standard benchmarks with no added tricks, get worse with scale on specific visual skills? This matters because vision-language models now have to do joint work — look at images, relate multiple views, read diagrams, ignore plausible but wrong answer choices. A larger language backbone does not automatically mean better eyes. If scale regresses on spatial perception or diagram reading, then picking “the biggest model” for deployment is unsafe, and provider reports that only show aggregate numbers will never reveal it. The authors even point to examples already visible in the Qwen3-VL technical report, such as the 8B model beating the 32B and 30B-A3B models on BLINK, that were reported but not analyzed as capability regressions.

The key idea, in plain words

Instead of asking “which family is best on average,” ask “on this specific skill, does the smaller sibling beat the larger sibling on the same questions, question by question?”

Imagine two students from the same school taking the same 150-piece puzzle test. If the younger student gets 45 puzzles right that the older student misses, while the older student only gets 9 that the younger misses, the younger has a net advantage of 36. That is not noise or a tie. It is a localized win.

The authors apply that logic inside each model family. A family means models released under the same name, which roughly share architecture and training approach but differ in size and often in details like vision encoder, training data mix, or alignment. They compare every smaller-larger pair in a family on every subtask defined by the benchmark itself — for example, Jigsaw puzzles assembled from image patches, deciding if a camera moved left or right between two frames, telling a real photo from an AI-generated one, or answering college organic chemistry with figures.

When the same subtask shows this small-beats-large pattern across independently built families from different labs, it is harder to dismiss as one lab’s training accident. It looks task-driven: something about that kind of visual work gets harder, not easier, as these current recipes scale up.

How it actually works, step by step

First, choose models and tests. Four open-weight families with at least two sizes each: six Qwen3-VL models from 2B to 235B-A22B, eight InternVL3.5 models from 1B to 241B-A28B, three Gemma 3 models from 4B to 27B, and two Molmo 2 models at 4B and 8B. “A22B” style names mean Mixture-of-Experts, where only a fraction of parameters are active per token — for example Qwen 30B-A3B activates about 3B parameters. Nine evaluation splits cover perception and reasoning, diagrams, K-12 science, multi-image perception, general multimodal skill, expert knowledge, and math with visuals: MME, AI2D, ScienceQA filtered to 2,017 image questions, BLINK, MMT-Bench, MMBench, MMMU dev and test, and MathVista.

Second, test identically and fairly. All models answer zero-shot with direct prompting and greedy decoding, with thinking modes turned off to separate model size from extra test-time reasoning. Multiple-choice prompts ask for only the option letter. Answers are checked by deterministic letter and numeric matching, with a multimodal fallback judge used for only 0.4% of responses.

Third, do per-question flip analysis. For a smaller model and larger model on one subtask, count two kinds of disagreement. Flips favoring the small model are questions the small gets right and the large gets wrong. Flips favoring the large are the reverse. The difference is the net advantage for the small model. A case is only reported if the subtask has at least 20 shared questions and the net advantage is at least 5. That filters out tiny subtasks and trivial asymmetries.

Fourth, stress-test the finding. They repeat the count on dense-only pairs to rule out that it is just a Mixture-of-Experts artifact. They require the larger model to have at least twice the total or active parameters. They check sensitivity to answer-option order by re-running the six strongest BLINK cases under reversed and rotated orders. They compute exploratory McNemar p-values for asymmetry, with the explicit warning that these are descriptive because cases were selected from the same data. They re-evaluate both models in every case with chain-of-thought prompting to see if extra reasoning fixes the losses. And they have both AI judges and three human authors label a sample of 1,300 flips as genuine errors versus bad benchmark annotation or formatting mismatch.

What they showed — results, honestly, with limitations

Aggregate scores still favor scale, but with clear exceptions. The largest model in each family is best or near-best on most benchmarks, yet Qwen 30B-A3B scores 62.4% on BLINK while the smaller dense 4B scores 66.4%, and Molmo 4B beats Molmo 8B on both BLINK, 63.5% versus 59.3%, and MMBench, 88.3% versus 87.9%.

At the subtask level the picture changes. They find 375 inverse-scaling cases across all four families and every benchmark. Raw counts are highest for MMMU with 115, BLINK with 95, and MMT-Bench with 87, and lowest for AI2D with 9 and MathVista with 3. Normalized by number of comparisons, BLINK leads at 14.4%, followed by MMT-Bench at 5.8%, MME at 5.3%, and MMBench at 5.0%. Rates by family are similar for Qwen at 3.7%, InternVL at 3.3%, and Gemma at 3.9%, with Molmo higher at 9.0% but from only one pair. Adjacent sizes alone, like 4B to 8B, still give 197 cases, so this is not only distant-size comparisons. About 87% of the affected family-subtask curves are non-monotonic, with ups and downs rather than a single drop, echoing U-shaped curves seen in language models.

The strongest individual cases are large. In BLINK Jigsaw, Qwen 4B beats 30B-A3B by 24.0 points with a net of 36, and beats 32B by 22.0 points. Molmo 4B beats 8B on Jigsaw by 20.7 points. In BLINK Forensic Detection, several dense InternVL models beat 30B-A3B with zero reverse flips — no question where the large model succeeds and the small fails. In ScienceQA Basic Economic Principles, Gemma 3 4B beats 27B by 39.4 points.

Crucially, 13 subtask-dataset pairs replicate in at least three families. Five are in BLINK: Multi-view Reasoning in all four families, plus Jigsaw, Forensic Detection, Object Localization, and Relative Reflectance. Others include MMBench Coarse Perception, MME Scene, MMT-Bench Autonomous Driving, Relation Reasoning and Emotion, and MMMU Investment, Organic Chemistry, and Thermodynamics.

These patterns survive strong controls. Dense-only analysis still yields 193 cases and 8 cross-family subtasks, including Multi-view Reasoning in all four. Requiring twice the total parameters leaves 326 of 375 cases; requiring twice the active parameters leaves 239 cases. In both restrictions 11 of 13 replicated subtasks survive. The six strongest BLINK cases stay positive under every deterministic option reordering. Chain-of-thought halves the problem but does not remove it: median net falls from +8 to 0, the larger model recovers 43.5% of questions it had lost, but 96 of 375 cases, or 25.6%, still meet the threshold, with persistence highest on MME at 48.6% and BLINK at 42.1% and lowest on MMMU test at 7.6%. Human validation suggests at least 84% of sampled flips are genuine model errors rather than bad annotation, with near-total agreement on visual reasoning benchmarks.

Limitations are stated clearly. This is observational, not causal: families differ in vision encoder, training data, alignment, and active parameters, so the work shows regressions among released variants, not a pure effect of parameter count alone. In fact the two larger families change encoder tier with size, though 72.3% of cases compare same-tier encoders and 10 of 13 replications survive on same-tier only. Only open models are studied, all questions are weighted equally, statistics are exploratory with only 35.5% of cases significant before correction and 16 significant after conservative correction over all 10,434 comparisons, and benchmark contamination or paraphrase sensitivity cannot be ruled out.

Why this might matter to me

For your interests in spatial reasoning, world models, text-to-CAD and parametric geometry, and 3D generation, this paper is directly cautionary.

The exact skills that regress most — inferring camera motion across frames, assembling jigsaw pieces, judging which surface is darker independent of shading, localizing objects, matching points across views, and detecting real versus generated images — are the building blocks of spatial intelligence. A world model that cannot keep left versus right consistent between two frames, or that misplaces a point labeled on a wall onto a sofa and then reasons correctly from that false premise, will not maintain a coherent scene representation over time. One qualitative example shows exactly that: the largest Qwen model puts point A on the couch instead of the wall and concludes the colors are the same, while the smaller 32B model locates both points correctly.

For text-to-CAD and parametric work, the lesson is sharper. Those tasks demand exact reading of values, constraints, and geometry from drawings and tables, not plausible-sounding guesses. The paper finds larger models can be more drawn to plausible distractors — for instance picking a near-correct investment return like 27.39% instead of computing 28.57% from prices, or misreading traffic light color as yellow instead of red. In CAD that kind of confident distractor error becomes an invalid sketch, a flipped constraint, or an unsafe part.

Practically, it argues for task-aware scaling in your stack: do not assume a 30B or 200B+ variant will preserve the spatial competence of its 4B or 8B sibling. Evaluate per capability — multi-view consistency, correspondence, reflectance and depth invariance, diagram grounding — rather than by aggregate benchmark score, keep small models as baselines for geometry-heavy checks, and treat chain-of-thought as partial mitigation rather than a fix, since almost half of BLINK and MME regressions persist with reasoning enabled.

Terms worth knowing