The one-paragraph version
This paper asks a landscape-level question: how much of computer vision can now be done through a chat-style, general-purpose AI system that takes images and language instructions, and where do dedicated vision models still win? The authors test six frontier systems — GPT-6 Astra plus Fable 5, Kimi K3, Gemini 3.1 Pro, Qwen 3.8-Max, and Muse Spark 1.3 — across 34 capabilities and 55 existing benchmarks covering recognition, reasoning, spatial understanding, 2D and 3D prediction, video, generation, robotics, and expert domains like medicine and remote sensing. They compare each generalist against dedicated specialists, meaning models built and trained for one task such as detecting boxes or estimating depth, and against reported human scores where available. The pattern is consistent: Astra leads overall and generalists now match or beat references on semantic and object-centered tasks like reading documents and charts, math reasoning with images, 2D detection and segmentation, and 3D grounding by naming and boxing an object. They still lag badly where the output must be metrically exact, pixel-faithful, consistent over time, or require rare expert knowledge — examples are depth estimation, 3D reconstruction, image restoration, video segmentation boundaries, and pathology cell typing. Extra reasoning steps and calling specialist tools help on some tasks but not all, and often at much higher compute cost.
The problem they're solving, and why it matters
Computer vision used to advance one specialist at a time: one model family for classifying images, another for finding boxes around objects, another for marking every pixel belonging to an object, another for estimating distance per pixel, and so on. Now frontier general-purpose systems, meaning large vision-and-language models that accept natural-language instructions plus images or video and can output text, boxes, masks, or actions, claim to do many of those jobs through one interface.
The problem is that evidence about these claims is scattered. One benchmark shows a generalist beating another generalist, but not whether either is close to a specialist or to people. A strength on one task may not transfer to a related task. For the field, the practical question is where a generalist is now good enough, where headroom remains, and where future research on dedicated models will matter most. This matters because if semantic understanding is largely solved through a general interface, effort can shift to what the authors call perceptual precision: getting geometry, fidelity, and temporal consistency exactly right.
The key idea, in plain words
The key idea is to map maturity, not just rank models. Instead of declaring one winner, the authors organize vision into nine areas and 34 capabilities, test all six systems on the same samples with the same instructions, and then place the best generalist as a percentage of the best available reference for that capability.
Think of four buckets. Exceeds reference level means the best generalist scores more than 110 percent of the reference. At reference level means roughly tied, 100 to 110 percent. Approaching means 85 to 100 percent. Substantial gap means below 85 percent. The reference itself can be a specialist model, an earlier generalist best score, or human performance, depending on what exists for that benchmark. The authors' thesis that emerges is simple to state in words: generalists are increasingly good when vision can be interpreted, reasoned about, or organized around objects. They are still weak when vision must be measured or faithfully rebuilt.
A concrete contrast from the paper makes this clear. Astra can correctly find the blackboard described in words in a 3D room and draw a sensible 3D box around it, beating the tested specialist. But its depth map for a scene, meaning its guess of distance for every pixel, captures the rough layout while smoothing over edges and surface detail.
How it actually works, step by step
First, define the territory. The nine areas are recognition and visual reading, visual reasoning, spatial reasoning, 2D grounding and segmentation, 3D perception and geometry, video understanding, generation plus editing plus restoration, robotics and driving, and expert domains. A capability here means a grouped skill, for example visual counting or 2D spatial reasoning. A benchmark here means an existing test set with a defined scoring rule.
Second, run the generalists. All six systems get the same task instructions, visual inputs, and samples. Where a system offers different reasoning strengths, the authors use the strongest appropriate setting. Outputs are converted into the form the benchmark expects. For example, predicted outlines are turned into pixel masks for segmentation scoring, predicted surfaces and anchor depths are rendered into dense depth maps, and predicted centers plus sizes plus rotations are turned into 3D boxes. For generation and editing, the reasoning model does not draw pixels itself; it writes instructions for a paired image model — Astra is paired with GPT Image 2.5 Sunburst, and the other systems have their own paired image models.
Third, add references. For each capability the authors report a specialist score and, when published, a human score. Table 2 in the paper names the specialist used per capability, from Rex-Omni for 2D detection to Depth Anything 3 for depth to VGGT-1B for reconstruction to MIRAGE for restoration.
Fourth, ask four research questions in order. How broad is coverage relative to references? Where do generalists converge versus differ? Which task properties predict that specialist-level work is available through a general interface? Can extra reasoning effort, explicit tool use like pointing then counting, or a pipeline where a specialist acts as a tool for the generalist close the rest, and at what cost in inference compute?
What they showed — results, honestly, with limitations
Breadth is wide but uneven. In reading, all six systems are strong: text recognition scores range 93.7 to 98.1 compared with 98.0 for the human reference, and document, chart, and infographic understanding scores of 89.5 to 92.8 all beat the reported human score of 89.3. In reasoning, all six meet or beat the human score on visual math reasoning, 78.8 to 92.5 versus 78.7, with Astra at 92.5. Astra reaches human level on 2D spatial reasoning, 96.0 versus 95.8, and approaches it on 3D and multiview reasoning, 89.6 versus 94.1.
In 2D structured prediction, meaning outputs like boxes and masks rather than just words, Astra exceeds the tested specialist on detection by 10.7 points and on segmentation by 4.3 points, and its grounding score of 93.1 approaches the 96.0-range human reference. Qwen 3.8-Max also beats the specialist detector, so this is not only one model. Pose estimation, meaning locating body joints, is less mature: Astra at 75.2 versus 83.6 for the specialist, though far above other generalists at 19.6 to 52.5.
The split sharpens in 3D and video. All six generalists beat the tested specialist on 3D visual grounding, where the job is to find the described object in 3D, with Astra at 73.8 versus 46.5 for the specialist but still below 95.0 for humans — so “beats specialist” does not mean solved. Astra nearly ties the specialist on 3D object detection, 17.3 versus 16.8. But depth error remains much higher than the specialist, 0.63 versus 0.44 where lower is better, and multiview reconstruction error is about 2.3 times the specialist, 1.40 versus 0.61. Video understanding is competitive with the specialist, while video segmentation shows Astra at 84.5 approaching the specialist at 91.0 but other generalists far behind at 32.0 to 57.9.
Generation versus restoration is the starkest fidelity gap. Astra beats specialist references on text-to-image generation, instruction-guided editing, and quality assessment, but all generalists score only 17.2 to 17.7 decibels on restoration, a standard image-quality scale where higher means closer to the true image, versus 30.7 for the specialist. The paper’s low-light example is telling: the generalist output looks plausible but turns four bowling pins into five and changes text. Expert domains follow the same pattern: strong on general medical and satellite-image understanding, weak on fine-scale remote-sensing grounding at 26.4 versus 75.5, and very weak on microscopy and pathology typing at 14.1 to 23.8 versus 57.1 specialist and 82.0 human, even when nuclei are correctly located.
More reasoning and tools help selectively. Higher reasoning brings 2D spatial reasoning, 3D detection, and generation/editing to reference level in these tests, and tool use helps counting and medical grounding, but segmentation gains plateau. Costs vary: tool use adds only about 1.04 to 1.20 times the baseline cost, while maximum reasoning can cost up to about 13 times baseline for small gains.
Limitations to keep in mind: benchmarks were deliberately chosen to retain headroom, so this is not a random sample of all vision. The meaning of “reference” changes by row — sometimes specialist, sometimes prior generalist, sometimes human — which affects the maturity percentages. Some pairings, especially generation where a separate image model does the drawing, test a system rather than a single model. It is unclear in a few rows which number is human versus specialist, and human scores themselves depend on how they were collected.
Why this might matter to me
For spatial reasoning in vision-language models, the result is encouraging but bounded. Two-dimensional relations and multiview question answering look increasingly like a shared generalist strength, with Astra’s large margins over the next generalist — plus 13.6 on logical reasoning and plus 12.4 on 3D multiview — suggesting recent progress is real. For world models that need to predict what happens next in a physical scene, the caution is that high-level video understanding does not yet imply dense, temporally consistent prediction; boundaries, small objects, and nearby instances still drift across frames.
For text-to-CAD and parametric geometry, where a shape must have exact sizes, constraints, and topology rather than just look right, the depth and reconstruction gaps are directly relevant. An object-centered box or a language-grounded part reference may now be obtainable from a generalist, but metric distance, surface detail, camera alignment across views, and faithful recovery of edges remain specialist territory. The restoration hallucination — plausible pins that are the wrong count — is a useful mental model for why a pretty CAD-like shape can still be unmanufacturable.
For 3D generation and scene representation, the suggested division of labor is practical: let the generalist interpret the request, plan, and coordinate, and let specialists supply metric geometry, dense correspondence across views, or high-fidelity cleanup. The paper’s specialist-as-tool results, for example using a segmentation model to enumerate instances before counting or a depth model to compare distances, point toward hybrid pipelines rather than pure generalists for precision-sensitive 3D work.
Terms worth knowing
Frontier general-purpose system: a large AI model that handles many vision and language tasks through instructions, as opposed to one model per task.
Specialist model: a model whose architecture or training is designed for one capability, such as Rex-Omni for detection or Depth Anything 3 for depth.
Human reference: published human accuracy on the same benchmark, used to judge remaining headroom beyond specialists.
Grounding: finding the image or 3D region described in words, for example “the patchy consolidation in the mid left lung” or “the blackboard.” Often scored by overlap thresholds.
Detection versus segmentation versus pose: detection draws boxes around objects; segmentation marks every pixel belonging to each object; pose estimates joint positions for people or articulated shapes.
Overlap and accuracy scores: Intersection over Union, often shortened to IoU, means area of overlap divided by area of union for two regions, in words how much prediction and truth coincide. Mean Average Precision and J&F are aggregate scores that reward both finding the right instances and tracing clean boundaries over video frames.
Monocular depth and reconstruction: depth estimation guesses distance for every pixel from one or more views; reconstruction builds 3D points or surfaces in a shared coordinate system. Errors here are often reported as absolute relative error or point-cloud error where lower is better.
PSNR and SSIM: two fidelity scales for restoration that compare a recovered image to ground truth pixel by pixel and structurally; higher means more faithful, not just better looking.
Reasoning effort and tool use: spending more intermediate computation before answering, or calling an explicit helper such as a pointer, counter, depth model, or image generator, then verifying and completing the task.