World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories

The one-paragraph version

World Motion Models, or WMMs, propose a single universal language for describing a moving 3D world: everything interesting is approximated as a small set of rigid moving coordinate frames tracked over time. A human becomes a few dozen joint frames, a robot becomes its link frames, a rigid object or a camera becomes one frame, and even a robot action becomes a virtual target frame. Once everything is in that same format, the authors train one kind of generative model — a flow-matching transformer where every frame-at-a-time gets its own noise level — so the same network can answer any “given what is known, fill in what is unknown” question. Change which tokens you tell it and which you ask it to imagine, and the same weights act as a future predictor, a robot policy, a forward simulator for control, a motion in-filler, or a cross-body retargeter, plus extra out-of-time context tokens for distant goals or example pairs.

The problem they're solving, and why it matters

To act in the physical world, an agent needs a prior over “what was, is, and will be where across time”: how people will move, what a push will do, what actions reach a goal, what happened in between two keyframes.

Today that prior is fragmented. Human motion uses body-specific parameters, hands use a different hand model, robots use their own kinematic description files, and there is no shared model across actor types. The other dominant route — video world models — predicts changing pixels, which tangles true 3D motion together with appearance, lighting, and camera viewpoint. Dense 3D point tracks are better grounded but are translation-only and wasteful: thousands of points to describe what a few rigid joints could describe, with no notion of rotation that matters for contact and articulation.

WMMs ask: can we have one compact, physically grounded, category-free representation, and one flexible inference scheme that covers causal prediction and non-causal tasks like inverse dynamics, planning between waypoints, and retargeting between bodies?

The key idea, in plain words

The core observation is that real-world motion is largely low-rank and piecewise rigid. Think of a waving hand: the skin deforms densely, but the cause is a few rigid bones moving. The same holds for a robot arm, a box being pushed, or background motion that can be interpolated from a motion basis.

So the authors make rigid motion the primitive. Meaning of the key phrase: a rigid pose in 3D — position plus orientation — tracked over time. A person, robot, object, camera, or action is just a collection of such trajectories sharing the same space. No separate heads for separate morphologies; everything is a “pose proxy.”

Flexibility comes from how they corrupt and denoise. Instead of one global noise level for the whole sequence, as in a standard diffusion model, each small unit — one entity at one time — gets its own independent noise level during training. Some tokens are almost clean like observations, others are almost pure noise like targets to predict, in every random combination. At test time you simply declare: these tokens are known, those are unknown, and integrate the learned velocity field to fill in the unknown. A separate prefix of virtual context tokens, living on a different timeline with its own position coding, holds things that are not part of the continuous present — a far-future goal, an A-pose that labels “which joint is which,” or a human-robot example pair that says “this maps to that.”

How it actually works, step by step

1. Turn the world into pose-plus-description tokens. The elementary unit is “where plus what.” Where is a pose: 3 numbers for position plus a rotation encoded as two columns of a rotation matrix, packed into 9 numbers with a scaling factor to keep training stable. What is a feature that stays constant along a trajectory: a semantic part — either a visual feature queried from an image or a learnable tag like joint versus object versus action — plus an optional geometric part that summarizes local 3D shape with a small point-cloud network.

A concrete example from the tabletop pushing task: each block is one trajectory, the robot gripper current pose is another, the robot commanded target pose is another. Each carries a tag for what it is and a shape summary.

2. Group into a set of trajectories plus context. An entity over time is a sequence of its poses with its shared description. A scene is a set of such sequences — say M entities by N time steps. For non-causal anchors, each trajectory gets an extra prefix of virtual steps that are not on the physical clock — for example the final goal frame, or two paired demonstration frames showing a human pose next to the matching humanoid pose.

3. Train with independent noise per token. During training, sample a noise-level field with one value between 0 for clean and 1 for fully noisy per token, and a Gaussian noise field of the same shape. The input is an interpolation between the true pose and noise controlled by that level. The network is trained to predict the direction from clean to noise, with squared error summed only over tokens that have valid ground truth. Because the split between almost-clean and almost-noisy is random each batch, one objective covers essentially every “predict some from others” distribution. The authors note you can bias the schedule, e.g. often keep history clean and future noisy to sharpen forecasting.

4. Infer by choosing a mask. A trained model is a family of samplers. Set noise level to 0 for observed tokens and 1 for targets, then integrate from noisy to clean. Past known plus future unknown is prediction or, if action entities are included, a policy. Actions known plus future states unknown is a forward dynamics model. Sparse keyframes known plus rest unknown is planning and in-filling. One body known plus another body unknown is retargeting or partner prediction. Noisy real observations can be given a small level like 0.05 so the model treats them as hints and slightly corrects them.

5. Architecture for scale and missing data. A modest scene of 30 frames and 400 trajectories is already 12,000 tokens, too many for dense attention. The transformer therefore uses factored attention — separately along time, along entities, plus global and register paths — and extra learnable register tokens that add capacity cheaply. Conditioning on noise level, description, and language or image is injected per token with adaptive normalization in every block. Invalid steps from occlusion or tracker dropout are pinned to full noise, kept in attention but excluded from loss. For more entities or longer horizons than training saw, an anchored sliding-window sampler denoises chunks in stages while reusing already-denoised anchors to keep seams consistent.

What they showed — results, honestly, with limitations

The authors train a separate checkpoint per benchmark — not yet one joint model across all domains — and test versatility:

• Tabletop language-conditioned pushing: 5 task families, 50 rollouts each. WMM reaches 87.2% macro-average success, above the listed prior best of 80.0% and well above video-pretraining baselines. As a differentiable world model with gradient-plus-sampling predictive control toward an imagined goal, it improves absolute-position tasks from 90% to 100% and 76% to 80% and reaches success in fewer steps.

• Dense 3D future prediction from a single start frame: On two subsets with 400 points over 32 steps, WMM beats the prior method on all reported trajectory errors. Notably, training on 128 randomly sampled tracks and testing on 400 does better than training on 400 directly. The paper hypothesizes this prevents overfitting, which is plausible but not directly proven. Joint training on both domains works with only a small drop, suggesting cross-domain sharing is feasible.

• Object-conditioned human-object interaction synthesis over 120 frames: Roughly comparable or better than the specialized baseline: much lower start error and foot-height error, better contact precision-recall balance and joint-position error, but worse on horizontal path error, foot sliding, and object orientation by their table. Qualitatively it stays plausible where the baseline interpenetrates, and generates diverse samples. It is smaller in parameters but slower per sample in their timing.

• Human to humanoid retargeting evaluated in physics simulation: One checkpoint handles clean 30-frame-per-second input, sparse 3-frame-per-second keyframes to infill, and per-frame noisy input. Compared to a fast real-time baseline and a slow offline optimizer, WMM has highest throughput and highest sequence success rate and lowest local tracking error in all three regimes in their table.

• Extra demos: motion planning between start and goal across 8+ different arms in one model, and hand-object generation where either hand or object can condition the other.

Ablations support the design: removing per-token noise collapses policy success to zero; removing per-block conditioning drops success by 6 points; collapsing the factored register architecture to a plain transformer massively worsens trajectory error.

Stated limitations are important: one checkpoint per task, rigid-only with no scaling or shearing, requires depth and current/past state rather than raw RGB video, and does not generalize to entirely unseen morphologies. The video-to-3D-track labeling pipeline itself costs about two minutes per short clip in their measurement.

Why this might matter to me

• Spatial reasoning in LLMs/VLMs: If language-vision models are to reason about “slide the green star to the yellow star,” they need more than 2D pixel change — they need object poses, contact, and goals in 3D. WMMs offer a compact pose-token interface that a language prompt can condition, and that returns both an imagined goal frame and an action chunk — a natural bridge between linguistic goals and spatial rollouts.

• World models: Most world models are either causal video predictors or latent dynamics tied to one body. Here policy, predictor, and simulator are literally different masks of one flow field, plus goal tokens for model-predictive control. That is a useful template for a world model that plans, infills, and inverts without separate modules.

• Text-to-CAD and parametric geometry: You work with structured parameters, not pixels. WMMs still respect parametric structure — they directly animate meshes by skinning or rigid transforms and only project to joint angles or body parameters for scoring — but they model the shared pose space underneath. That suggests text-to-CAD could condition a shared motion/geometry prior, then project to valid parametric constraints, rather than learning isolated per-category generators.

• 3D generation and scene representation: Instead of dense motion maps or point flow, the sparse rigid-basis view says: generate a few dozen SE(3) trajectories, then interpolate, skin, or assign dense deformation. That is far sparser and rotation-aware, and it unifies cameras, objects, humans, and robots in one scene representation — directly relevant to 4D generation, dynamic reconstruction, and simulation-ready assets.

Terms worth knowing

• SE(3): the space of rigid poses in 3D — where something is plus how it is oriented, without stretching or shearing. Translation-only point tracks are its rotation-free special case.

• Trajectory / eigenrigid basis: a path of poses over time; the claim that many dense deformations are combinations of a few rigid paths.

• Flow matching: a generative method that learns a velocity field pointing from noise to data along straight-line interpolations, then integrates it to sample.

• Per-token noise level / Diffusion Forcing: giving each entity-time its own corruption amount so one network sees all known-unknown splits. Zero means observed, one means generate.

• Any-to-any marginal: shorthand for “any subset can condition any other subset” — prediction, policy, planning, and retargeting as different masks.

• Context tokens: extra virtual-time tokens outside the current episode for goals, identity templates, or cross-body example pairs, with separate position codes.

• AdaLN / factored attention / registers: implementation tricks — per-token scale-and-shift conditioning; attending separately over time and over entities for efficiency; extra learnable tokens that add capacity cheaply.

• SMPL, MANO, URDF: standard parametric descriptions for human bodies, hands, and robots that WMMs generalize beyond by treating everything as generic pose proxies.

• Model-predictive control (MPC), inverse dynamics, retargeting: using a forward model to optimize actions toward a goal; inferring actions from states; mapping motion from one body like a human to another like a humanoid.