All summaries · 2026-10-31

Decoding Geometry, Not Tokens: Efficient VLMs for Robot Affordance and Trajectory Prediction

Akila Ayanthi, Darshana Priyasad, Tharindu Fernando, et al.

arXiv:2610.39416 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-10-31

The one-paragraph version

This paper argues that small vision-language models already understand space, but we ask them to show it in the wrong way. Today a robot that needs to know where to grab and how to move typically gets those answers by making a language model write out numbers as text, one digit at a time — for example "x equals 123, y equals 456." That text interface throws away precision and lets small errors snowball. The authors instead freeze a small, off-the-shelf model — meaning they never change its internal weights — and attach a tiny new output module that reads the model's internal activity directly and predicts continuous geometry in one step: a rectangle for the graspable region and a short list of points for the gripper path. With only about 4 million new parameters trained, their 0.5 to 2 billion-parameter systems cut trajectory shape error by more than two-thirds, multiply affordance accuracy several-fold, and come close to a much larger robotics specialist called RoboBrain while running about 3.8 times faster.

The problem they're solving, and why it matters

Robot manipulation needs two kinds of geometric answers before acting. The first is affordance prediction, which in plain words means: given an instruction like "reach for the spoon," which image region can you use to do it? The paper treats this as a rectangle, or bounding box. The second is trajectory prediction, which means: what path should the gripper follow through the image? The paper treats this as up to four ordered points, or waypoints.

Standard vision-language models, or VLMs — systems that take an image plus text and generate text — handle these geometry tasks by turning coordinates into language tokens. A token here means a chunk of text from a fixed vocabulary, like a word piece or a digit. Autoregressive decoding means generating those tokens one after another, where each new token depends on the previous ones.

The authors say this is structurally mismatched for three reasons they explain in words: the training goal rewards picking the right word, not being geometrically close; the fixed vocabulary forces continuous positions into coarse integer bins, creating rounding or quantization error; and because tokens are made in sequence, a mistake in an early digit corrupts later digits.

The practical stakes are compute. Large embodied models work better partly because they are big and trained on huge robot datasets, but they are too slow and heavy for robots with limited onboard hardware. The question is: can a compact, already-available VLM of 0.5 to 2 billion parameters give accurate 2D geometry while staying fast, if we only fix how we read geometry out?

The key idea, in plain words

The key idea is: don't teach the small model new spatial knowledge — expose the spatial knowledge it already has, with a different plug.

Think of the frozen VLM as a librarian who has already read the image and instruction together and taken rich notes on every layer of scratch paper. Asking the librarian to dictate coordinates digit-by-digit over the phone garbles the message. Instead, hand those notes to a skilled draftsperson who draws the rectangle and path directly with a continuous pencil, without rewriting the notes.

Concretely, that means: keep the entire VLM frozen, tap its intermediate internal activity at one chosen depth, use the instruction summary to point attention at the right image area, and then directly estimate numbers between zero and one for box corners and path points. No text generation, no digit-by-digit loop, no changes to the backbone. The contribution is not inventing the draftsperson architecture — it borrows a style called DETR, described below — but showing that the decoding interface was the bottleneck, and that frozen compact models already hold usable geometry.

How it actually works, step by step

Imagine the input is a photo of a table plus the instruction "drag mug to the right."

1. Run the frozen backbone once, and stop early. A backbone here means the pre-existing VLM used as a feature extractor. An image encoder turns image patches into visual features, a text tokenizer turns the instruction into text features, and a large language-model stack mixes them together layer by layer. A hidden state means the internal numerical activity at a given layer; a hidden layer is one step in that stack. The authors do not run to the final layer. They stop at a chosen intermediate layer, called l-star, which is picked separately per model and task based on validation. For example, for trajectory they use layer 13 for Qwen2-VL-2B and layer 20 for LLaVA-OneVision-0.5B; for affordance they use layer 12 and 23 respectively. Stopping early is why they save compute.

2. Build a task-pointed starting question. They take two things from that intermediate layer: the visual token memories, which are the refined image-patch descriptions, and the hidden activity at the final instruction word, which has looked at the whole image-plus-text context and therefore summarizes "what task, in what scene." A learned starting query — think of it as a blank sticky note that says "find the relevant place" — is shifted by that instruction summary plus a learned position bias. In words: from the first step, the query is already steered toward instruction-relevant regions rather than searching blindly.

3. Refine by looking at the image. That query goes through a small transformer decoder of 3 layers, width 256, 8 attention heads. Cross-attention here means the query repeatedly asks: which visual memories are most relevant, and pulls their information in. There are two independent copies of this decoder, one for the box and one for the trajectory, so each can learn its own looking pattern.

4. Draw continuous outputs in one forward pass. A small three-layer neural network, called a feed-forward network, maps the refined query directly to numbers. The affordance head outputs four numbers for left, top, right, bottom of the box. The trajectory head outputs all waypoints at once. A squashing function constrains each to the zero-to-one range relative to image width and height, later multiplied back to pixels. This is regression, meaning direct estimation of continuous values, not picking words.

5. Train only the small head with geometry-aware penalties. The backbone never receives gradients. For boxes, the loss combines point-wise closeness on corners — a smoothed absolute error that is extra sensitive to small errors — plus a term based on Generalized Intersection over Union, which in words penalizes both poor overlap and loose enclosing area. For trajectories, the loss combines smoothed error over all points plus extra weight on the first and last points, because where contact starts and where manipulation ends matter most. Training took about 4 hours on one high-end graphics processor in the authors' setup, versus about 15 hours for their comparison fine-tuning approach.

What they showed — results, honestly, with limitations

They train the head on ShareRobot data and test trajectory on ShareRobot-Bench-Trajectory, affordance on ShareRobot-Bench-Affordance and on AGD20K, where AGD20K is an out-of-distribution set of everyday objects with boxes constructed from masks. They report shape error with Discrete Frechet Distance, or DFD, which cares about order-sensitive path shape with lower being better; Hausdorff Distance, or HD, which captures the worst single gap; Root Mean Squared Error, or RMSE, which averages point errors; and mean Average Precision, or mAP, which averages box correctness across overlap thresholds with higher being better.

For trajectory, the gains are large and consistent. Changing Qwen2-VL-2B from text-token output to their geometric head drops DFD from 0.798 to 0.251, with uncertainty about plus or minus 0.010. Changing LLaVA-OV-0.5B drops DFD from 0.777 to 0.225, plus or minus 0.039. That is more than 68% reduction, with similar drops in HD and RMSE. Their 2B model at 0.251 DFD approaches RoboBrain at 0.191, while running 21.0 versus 5.5 frames per second on an H100 and using 1.2 versus 31.0 trillion floating-point operations per sample in their measurement protocol.

For affordance, gains are also strong but more uneven. Qwen2-VL-2B goes from 7.30 to 12.37 mAP on AGD20K and from 6.27 to 11.58 on ShareRobot-Bench, roughly a 1.7 times increase and about an 85% increase on the latter. LLaVA-OV-0.5B jumps from 1.09 to 9.53 on AGD20K, about an 8.7-fold increase. This brings small models near or past larger 7B zero-shot baselines that score for example 18.79 and 11.87 on AGD20K.

Their ablations, meaning controlled removals, disentangle why. Simply replacing text decoding with a simple continuous estimator on the final layer already captures about 91.6% of the full trajectory gain for Qwen and 86.8% for LLaVA. Fine-tuning with a method called LoRA, or Low-Rank Adaptation — small trainable add-ons to the language model that keep text decoding — barely helps trajectory and sometimes hurts affordance, staying far behind the geometric head. For affordance on Qwen, depth and the attention-based decoder add substantially more on top of the interface change. A layer sweep shows every tested intermediate layer beats text decoding for trajectory, while affordance varies more with depth. Even features before the language stack already predict well, suggesting the language layers reorganize rather than create spatial signal.

Limitations, many stated by the authors: they use separate heads for box and path rather than one unified decoder that could exploit their relationship; early exit saves active computation but still requires storing weights up to the exit layer; and evaluation is only instruction-conditioned 2D prediction before execution, not 3D or closed-loop control with real manipulation success. Throughput numbers are also protocol-specific and model-specific because different models skip different fractions of computation.

Why this might matter to me

If you care about spatial reasoning in LLMs and VLMs, this is direct evidence that failure to output coordinates as text does not mean the model lacks spatial representation. The bottleneck can be the readout, not the world knowledge. That suggests probing intermediate activations and designing continuous readouts before scaling up or retraining.

If you care about world models — internal models that predict how scenes evolve under actions — trajectory waypoints are a minimal 2D world-model output: start, middle, end conditioned on language. The finding that endpoint supervision prevents collapse to a dataset-average path is a useful lesson: without anchoring where contact begins and ends, a smooth average error will happily predict a blurry middle.

If you care about text-to-CAD and parametric geometry, the message transfers cleanly. CAD wants precise continuous parameters like line endpoints, arc centers, extrusion depths, not digit tokens. A frozen language-vision backbone plus a small parametric head that regresses numbers with geometric losses like overlap and endpoint penalties is a template for turning vague language plus an image or sketch into exact parameters, while keeping inference cheap.

If you care about 3D generation and scene representation, think of the box and 2D path as the simplest scene representation: where is the actionable object, and where will the actor go. The same pattern — condition a query on language, cross-attend to visual or 3D tokens like views, point clouds, or neural field features, then regress continuous poses, boxes, or splats — could be a fast alternative to tokenized 3D coordinates. The open step the paper does not take is lifting this to depth, 6-degree-of-freedom poses, and closed-loop use.

Terms worth knowing