GeoWM: Efficient Direct World Modeling in Explicit Geometry

The one-paragraph version

GeoWM is a world model that predicts future 3D shape directly, instead of predicting future video frames or abstract visual features and then trying to recover shape afterward. It starts from five recent color images, uses a frozen geometry foundation model — in this paper, VGGT-1B — to turn those images into depth maps and camera poses, which together form a geometric history. A tiny camera-motion predictor guesses where the camera will be at a requested future time, and the last observed depth is reprojected into that predicted view to make an anchor, meaning what the current static world would look like from the future viewpoint. A 206-million-parameter flow-matching diffusion transformer, conditioned on the depth history, the anchor, and the requested horizon, then predicts the future depth map in one shot, with no step-by-step rollout. Across urban driving (KITTI, Cityscapes), aerial flight (TartanAir), and robot manipulation (DOMINO), the authors report the lowest depth error and 3D Chamfer distance in every horizon group, plus the lowest camera-trajectory error on moving-camera datasets, while keeping inference time flat at about 0.11 seconds per horizon.

The problem they're solving, and why it matters

Understanding 3D structure and how it changes matters for driving, flying, and manipulating objects: a robot needs to know not just what it sees now, but what shape it will see in 0.5 or 4 seconds if it and other things keep moving.

Recent geometry foundation models are good observers — they reconstruct depth, camera pose, and point clouds from given images — but they do not forecast. World models do forecast, but the paper argues they forecast the wrong thing:

• Video world models like Cosmos forecast future color frames because they are trained to look photorealistic. Good-looking pixels do not guarantee correct 3D placement, and moving objects can be mislocalized.
• Latent feature models like DINO-Foresight forecast abstract features of a frozen vision encoder, then decode depth with a separate head. Decoding can amplify small feature errors.
• Even geometry-adjacent latent models like VGGT-World forecast internal tokens of a geometry model and decode them with pretrained heads, rather than forecasting depth itself.

Almost all of these extend to long horizons by autoregressive rollout, meaning feeding each prediction back in as input for the next step. That compounds errors and makes cost grow with distance into the future. The paper asks: what if the state of the world model is explicit geometry, and each future time is queried directly and independently?

The key idea, in plain words

Think of forecasting as answering: “From where I will be in h seconds, what depths will I see?”

GeoWM makes the horizon — how far ahead you want — an input to the model, like dialing a time knob, not a number of steps to simulate. Because predicting far ahead from scratch is hard, it gives the predictor two crutches:

1. A geometric history: Instead of remembering colors, remember shapes. Five past frames become five depth maps, where a depth map is simply per-pixel distance from the camera, plus how the camera moved between them.

2. A viewpoint-compensated anchor: Guess the future camera motion with a very small regressor, then mathematically project the last depth map forward into that guessed future view. That warped image is not the answer — objects may move, occlusions appear, depth and motion estimates are noisy — but it shows roughly where static walls, roads, and furniture should land. The network only has to learn the correction for dynamics and newly visible regions.

Concrete example: driving forward, buildings shift outward in the image as you approach. The anchor already shifts them according to predicted ego-motion. The transformer then refines boundaries, fills in newly revealed side streets, and adjusts for a passing car that the anchor, which assumes a static world, cannot know.

How it actually works, step by step

Inputs and task. Take the K most recent frames, here K equals 5. A frozen reconstruction backbone returns a depth map per frame, shared camera intrinsics — the internal lens parameters that map 3D rays to pixels — and camera poses — position plus orientation. The task is: given a horizon h from a dataset-defined set H, predict the depth map as seen from the camera at time t+h, and report the relative pose from time t to t+h, which is the rigid transform that moves points from the current camera frame into the future one.

The authors predict depth rather than full 3D point maps because, once you know the pixel and lens, the viewing ray is fixed; only distance along that ray is unknown. Pose plus depth gives the full point cloud.

Camera-motion predictor and anchor. The K minus 1 observed relative motions are encoded as rotation vectors and normalized translations. A lightweight predictor — ridge regression for rotation and a small regressor for translation, at most tens of thousands of parameters — maps this history directly to all future poses in H at once, with no rollout.

Given the predicted pose and intrinsics, the last depth map is forward-projected into the future view with nearest-depth handling for occlusions. The paper writes this warped result as the anchor. For a static camera like DOMINO’s head camera, the anchor would just equal the last depth, so it is omitted there.

Forecasting network. The forecaster is a plain diffusion transformer operating directly on depth maps at 256 by 256 resolution, with no learned image compressor or tokenizer. Its spatial input is channel-stacked: K history depths plus the anchor plus a noisy target, split into 16 by 16 patches to make 256 tokens, then 14 transformer blocks of width 896, then unpatchified back to a depth map.

Conditioning is twofold: spatial conditioning by concatenation, and global conditioning by horizon and flow time injected into every block via adaptive layer norm with zero initialization. Flow time here means the noise level in flow matching, where 0 is pure noise and 1 is clean data. Both log-horizon and flow time are sinusoidally embedded and fused by a small network.

Training uses flow matching with clean-target prediction and an inverse-depth-weighted absolute error in velocity space. In plain words: errors on nearby pixels count more than errors far away, so distant sky or far road does not drown out nearby cars or tabletops. Pixels without ground truth are masked, depth is capped at 80 meters driving and 100 meters TartanAir, and a fixed fraction of training samples are pure noise to practice one-step inference. Each training example pairs an observed window with a randomly drawn horizon, so the same network is supervised at all horizons from real history only.

Inference. At test time the model does a single forward pass from pure noise at flow time zero and averages four such draws with different noise, then undoes per-horizon normalization. No differential-equation integration is needed. Each horizon costs the same fixed number of evaluations and different horizons can be batched. The camera pose forecast is reported directly from the lightweight predictor.

What they showed — results, honestly, with limitations

The authors compare against copy-last — simply repeating the last geometry, which marks no forecasting — plus DINO-Foresight for reference on driving, Cosmos-3 run zero-shot with the same frozen backbone for readout, and VGGT-World using released checkpoints on KITTI and Cityscapes and author-retrained versions on TartanAir and DOMINO.

Depth. GeoWM has the lowest absolute relative error — mean absolute depth error divided by true depth, lower is better — in every horizon group on all four datasets. On KITTI it ties Cosmos at the first frame at about 6.6 versus 6.8, within what the authors note is seed variation, then leads 17.4 versus 21.2 at long horizons of about 2.3 to 4.1 seconds. On TartanAir the lead grows from 12.0 versus 13.6 at one frame to 40.1 versus 57.5 at 11 to 20 frames. On DOMINO in-domain it roughly halves error over all horizons, 9.6 versus 20.5 for Cosmos. The paper summarizes 4 to 58 percent lower short-horizon error and 7 to 48 percent lower long-horizon error versus the strongest comparable baseline.

3D and pose. Chamfer distance — here mean of accuracy and completeness after similarity alignment and downsampling, lower is better — is lowest for GeoWM in every group, with one tie against Cosmos at KITTI mid horizons. Over all horizons the margin over Cosmos ranges from about 3 percent on KITTI to 39 percent on DOMINO. Camera trajectory errors termed absolute trajectory error, relative translation error, and relative rotation error are also lowest for GeoWM on the three moving-camera datasets, with one tie on KITTI rotation.

Generalization. Trained on 23 DOMINO tasks and tested on seen versus unseen tasks and robots, GeoWM stays best in every cell. Relative to copy-last over all horizons, it cuts error by 54 percent in-domain, 45 percent on a novel robot, 37 percent on novel tasks, and 31 percent when both are novel. Limitations appear here: on novel tasks the long-horizon margin narrows to about one point, and Cosmos has higher inlier rate termed delta-1 at the longest horizon.

Ablations clarify why. Forecasting frozen depth-head tokens instead of depth, even with an added decoded-depth loss, is at least 1.8 points worse over all KITTI horizons while using twice the tokens. Removing depth conditioning is worse than copy-last. A plain copy without camera compensation hurts most at short range; dropping history hurts most at long range. Scaling the anchor’s translation away from the predicted value forms a V-shaped error curve with minimum at the prediction, suggesting the model truly uses the viewpoint. Rolling out the same weights recursively reaches 38.9 versus 17.4 direct at KITTI long horizons and ends 2.3 times worse on KITTI and 1.9 times on TartanAir. One-step readout beats multi-step Euler integration.

Cost. Latency is flat at 0.11 seconds for any horizon, totaling 12 teraflops for 20 horizons. VGGT-World grows from 1.9 to 36.9 seconds and 1116 teraflops via rollout; Cosmos needs 4.9 seconds even near-term and 9.3 seconds at horizon 20 because it denoises a whole 17-billion-parameter clip. That is over 300 times and over 80 times faster at the longest horizon, respectively, with the smallest forecaster.

Limitations to keep in mind: depth is median-scaled per image except DOMINO and clouds are similarity-aligned, which can hide absolute scale errors; Cosmos is evaluated zero-shot while GeoWM is trained per domain; VGGT-World on two datasets was retrained by the authors, not released; qualitative results still show smoothed object boundaries; and the model is not conditioned on robot actions — listed as future work along with joint motion and geometry prediction.

Why this might matter to me

• Spatial reasoning in LLMs/VLMs: If you want language or vision-language models to reason about “what will be where,” GeoWM suggests giving them explicit, horizon-queryable depth plus pose rather than pixels or frozen features. An anchor plus direct query is an interpretable intermediate state a VLM could read, critique, or edit in words before acting.

• World models: The direct-query trick breaks the usual horizon-equals-compute tradeoff. For planning you often need sparse futures — e.g., 1, 5, and 20 steps — without simulating everything between. GeoWM shows you can train one network on sampled horizons from real history and query them independently, avoiding train-on-real versus test-on-predicted drift that scheduled sampling or flow-forcing try to patch.

• Text-to-CAD and parametric geometry: Text-to-CAD needs exact distances, extrusions, and poses, not plausible textures. The finding that explicit depth beats token-space forecasting, and that a tiny motion model plus reprojection supplies most viewpoint handling, maps to CAD: predict parameters in their native geometric space, condition on a projected prior from current parameters and intended edit or viewpoint, and let a generative refiner fix residuals.

• 3D generation and scene representation: Operating directly on depth maps with inverse-depth weighting and per-horizon normalization is a template for scale-aware 3D diffusion without autoencoders. The anchor idea generalizes: warp current Gaussians, meshes, or depth into a guessed novel view as conditioning, then diffuse the future. The flat-latency result also matters for interactive scene generation where long-horizon previews must be cheap.

Terms worth knowing

• World model: A model that predicts future states of the environment from past observations, used for planning and control.
• Geometry foundation model: A large network pretrained to reconstruct 3D structure — depth, pose, point maps — from images. Here VGGT-1B is used frozen to turn color frames into geometry.
• Depth map: Per-pixel distance from the camera along each viewing ray. Combined with lens intrinsics and camera pose it yields a 3D point cloud.
• Point map / point cloud: Per-pixel 3D points or an unordered set of 3D points representing scene surface shape.
• Camera pose and intrinsics: Pose is where the camera is and where it looks; intrinsics are internal lens properties that define how 3D rays project to pixels.
• Anchor: The last observed depth reprojected into the predicted future viewpoint. A physics-based guess of static structure from the future camera, used only as conditioning.
• Autoregressive rollout: Extending predictions by feeding each forecast back as input for the next step. Simple but accumulates error and cost grows with horizon.
• Direct / horizon-conditioned prediction: The requested time offset is an input, so each future is predicted independently from real history without intermediate futures.
• Flow matching / diffusion transformer (DiT): A generative formulation that learns a velocity field morphing noise into data; a transformer implements the denoising. Here it predicts clean depth in one step from noise.
• AbsRel and delta-1: Absolute relative error, mean of absolute error over true depth, lower is better; delta-1, share of pixels within 25 percent of true depth, higher is better.
• Chamfer distance (CD): Average nearest-neighbor distance between predicted and true point clouds, capturing 3D shape fidelity after alignment.
• ATE, RTE, RRE: Absolute trajectory error, relative translation error, and relative rotation error for camera motion after alignment, all lower is better.