The one-paragraph version
Instead of plugging camera pixels directly into a language model, this paper keeps vision completely outside the language model. A frozen set of detectors finds campus objects like benches, bollards and bikes and estimates how far away and in which direction each one lies. A small deterministic program — not a learned model — then compiles those estimates into short decision-ready text, for example "Bollard: 5 to 15 meters away, on the left side," plus counts and closest-object picks. An unmodified text-only language model, called the reader, reads only that text and answers multiple-choice questions about the scene. On a carefully matched campus-robot benchmark where both routes see the same visible scene including perception mistakes, this text route beat a same-size vision-language model that sees the raw image: 0.7892 versus 0.7462 accuracy at the 7-billion-parameter size, and 0.7673 versus 0.6913 at the 3-billion size. The authors show the win comes mostly from pre-computing the answer vocabulary in text, holds up when questions are reworded, fades when questions need brand-new judgments, holds down to a 1.5-billion reader but collapses at 0.5 billion, and disappears once both sides get the same task training.
The problem they're solving, and why it matters
A standard vision-language model works by turning image patches into extra input tokens, meaning word-like pieces the language model must interpret, using an image encoder, meaning a vision network that converts pixels to those tokens, wired directly to the language generator. That coupling means when you shrink the language model for a robot, drone or phone, visual grounding and language reasoning degrade together because both jobs share the same shrunken weights. The authors argue this hurts most exactly where small models are needed, and where current models are already known to be weak at counting, fine perception and spatial grounding. Earlier comparisons of image-input versus text-description were also unfair twice over: the text often described a privileged cleaner subset than the image showed, and the text wording could leak the exact answer strings. Without fixing scope and leakage, you cannot tell whether text wins because the format helps reasoning or merely because it cheats.
The key idea, in plain words
Free the reader from seeing. Let specialized vision modules do all the seeing, let plain code do all the measuring and sorting, and let the small language model spend all of its capacity on language. The principle the authors state is: compute what can be computed, and leave the reader only what is genuinely ambiguous. In practice that means code decides distance bands like under 5 meters, 5 to 15 meters, over 15 meters and side sectors like left, center, right from the estimated range and bearing, meaning distance and angle from the robot, rather than forcing the language model to do arithmetic from raw numbers like 4.2 meters. It also means code proposes a best guess plus a runner-up for closest object, because the closed-form pick alone is right only about 0.61 of the time, and splits counts into confident versus uncertain so the reader can weigh doubtful detections. Everything before the language model is frozen at test time, so no pixel or visual token ever enters it.
How it actually works, step by step
The testbed is the UT Campus Object Dataset, meaning front-camera images from a campus robot with three-dimensional boxes in robot coordinates. For each frame the authors generate one three-choice question from five families: counting how many of a type are visible, direction where the nearest of a type sits, nearest-class which type is closest, nearest-distance which distance band the nearest of a type falls in, and path-object what lies closest to the forward path. Questions are only asked about objects in the camera field of view within 40 meters, so the image route is never asked about what it cannot see.
Perception starts from the front stereo pair. Two tuned YOLO11 detectors, meaning fast box-finding networks trained per fold only on other recordings, are ensembled with a confidence rule that accepts a box if either detector is strong alone or both are weakly agreeable. Tiling recovers small objects, while an open-vocabulary detector is demoted to finding at most two missed objects of a kind only if none were found, because naive fusion flooded scenes with up to 86 phantom benches. Range comes from the median of four cues: ground-plane geometry from camera height, stereo disparity from the two cameras, class-height prior from typical object size, and learned monocular depth. Bearing comes from a fixed camera-geometry mapping, nothing learned. Boxes that agree in class and are within 2 degrees and a 1.3 times range ratio are merged.
The serializer, meaning the code that turns that object list into text, emits three kinds of fields. It writes a nearest-of-each-type line already banded into exact option wording, count lines split by confident versus uncertain tag, and borderline flags near a boundary whose width is set from the measured 68th-percentile range error so the flag tracks real sensor uncertainty. Crucially the stream is scoped to the question family known at deployment: only counting questions get the long per-object rows, cutting the median text from about 1842 characters to about 720 for other families.
Evaluation discipline is unusually strict. There is a development bank of 1366 questions for building, and a confirmatory bank of 1328 later questions each at least 1.5 seconds from any development frame. Detectors and calibrations use sequence-rotating folds, meaning every question is answered by a model that never saw its own recording. The win criterion, a paired accuracy difference with a cluster-bootstrap uncertainty range resampling the 21 recordings 20,000 times, was frozen before seeing the confirmatory bank. Reworded-question and novel-question probe banks were also frozen in advance.
What they showed — results, honestly, with limitations
Zero-shot, meaning no training on this task, the shipped scoped text system beat the same-family image system at both pre-specified sizes, with uncertainty ranges excluding zero. At 7 billion the gap was plus 0.0429, and at 3 billion plus 0.0761. It replicated on a newer family at 4 billion by plus 0.0595, but was unresolved at 8 billion. The gain concentrated where code can precompute the answer: nearest-distance by plus 0.2362 at 3B and plus 0.1890 at 7B, and counting by plus 0.1358 at both sizes. It lost on nearest-class at 7B by minus 0.0617, while direction and path-object were unresolved.
Three probes explain the mechanism. Rewording every stem and option left the advantage intact, so it is not wording leakage. Replacing pre-banded phrases with raw numbers removed 71 percent of the gain at 3B and 83 percent at 7B, so boundary-aligned precomputation dominates. On 3974 novel questions needing new 7 and 22 meter bands, cross-class ordinal comparisons and count-within-radius, the aligned serializer kept only a small edge at 7B and none at 3B, and the numeric version lost at 3B. In short, the interface is task-aware and bounded by the decision vocabulary it knew in advance.
Scale and supervision locate the floor and ceiling. The text route beat comparable 1.4B and 3.8B image models, collapsed at a 0.5B reader, and the ground-truth oracle, meaning text built from perfect human labels instead of detections, explains why: even with perfect perception 0.5B reaches only 0.6152 versus 0.8660 at 1.5B and 0.9699 at 7B. Above about 3B the limit is perception, not reading. When the image model gets low-rank adaptation, meaning fine-tuning via a small set of added weights, on 16,000 task examples it overtakes the zero-shot text system at 0.8441, but an equally trained text reader ties it at 0.8396 with no resolved difference. Both remain well below the 0.9247 ground-truth ceiling, so both are perception-bound. Learned scorers over the compiled state reach 0.8245 without a language model but collapse to chance on reworded and novel banks, while the zero-shot reader retains robustness and reaches 0.5860 on novel questions at 7B.
Limitations are stated plainly. This is one campus domain with temporal holdout within the same 21 recordings, not external validation. Real detectors were tuned in-domain per fold while the baseline image model was zero-shot, so the headline is system-level, not pure sensor parity — the text side also used stereo while the image side got the monocular keyframe. Total compute is not smaller: perception totals about 0.63 billion parameters and about 0.95 seconds per frame, comparable to the image model's vision tower, though per-question answering is faster at 0.196 to 0.241 seconds versus about 1.1 seconds, amortized over many questions per frame.
Why this might matter to me
For spatial reasoning in language and vision models, the lesson is concrete: small readers fail less because they cannot reason and more because they are forced to do threshold arithmetic from noisy numbers like 4.2 meters; handing them 5 to 15 meters away, on the left side plus an explicit uncertain tag converts a fragile spatial calculation into a robust symbolic match. For world models, the paper is a working example of a compact, auditable textual world state — class, range, bearing, confidence, closest and next-closest — that supports counting, nearest and side queries without learned visual tokens, and whose failure modes split cleanly into missed classes and ordering errors rather than opaque geometry errors. For text-to-CAD and parametric geometry, the analogy is direct: just as banding at 5 and 15 meters and sectorizing at plus or minus 15 degrees aligned computation to the decision vocabulary here, snapping continuous estimates to named parameters, constraints and tolerances before language reasoning may help small models more than giving them raw floats. For 3D generation and scene representation, it suggests a detachable path where heavy perception and depth live outside the language model, can run once per frame and be cached across many queries, and can be swapped without retraining the reader — useful when you want language to query a scene graph rather than re-encode pixels every time — but also warns that any such serializer inherits a vocabulary boundary and will need fallback numeric reasoning for open-ended prompts.
Terms worth knowing
Reader: the text-only language model that answers from serialized text and never sees pixels. Serializer or semantic serialization: deterministic code that compiles detections into short text with precomputed bands, sides, counts and picks. Pixel arm: baseline where a vision-language model receives the raw image plus question. Typed or Scoped arm: test system where the reader receives scoped serialized text plus question; Flat is the earlier unscoped full-list version. Oracle: same text pipeline built from human ground-truth boxes, showing the ceiling if perception were perfect. Blind arm: reader given question and options but no scene, measuring what wording alone gives away, about 0.41 versus 0.333 chance. Zero-shot: using the model exactly as released with no training on this data. Fine-tuning with LoRA: updating only a small added set of weights on 16,000 examples; QLoRA is its memory-saving quantized form. Paired difference and resolved: accuracy of one arm minus the other on the same questions; resolved means the 95 percent uncertainty range from resampling recordings excludes zero. Perception-bound versus reader-bound: accuracy limited by detector quality even with a strong reader, versus limited by reader capability even with perfect perception.